Conflicts of Interest in Public Policy Research

In this essay, I discuss the difficulty of sustaining an inquisitorial system of policy research and analysis when it is embedded in a broader adversarial political setting. Conflicts of interest in public policy research exist on a continuum from blatant pecuniary bias to more subtle ideological bias. Because these biases are only partially susceptible to correction through individual effort and existing institutional practices (peer review, replication), I consider whether a more explicitly adversarial system might be preferable to the awkward hybrid that exists today. But there are important disanalogies between policy-relevant empirical debates and the kinds of conflicts we address with our adversarial legal system. If we are stuck with a muddled inquisitorial-adversarial hybrid, we need to encourage norms of “heterogeneous inquisitorialism,” in which investigators strive for within-study hypothesis competition and greater clarity about roles, facts, and values. THE VARIETIES OF CONFLICT OF INTEREST In public policy research, as in other domains of professional life, conflicts of interest (henceforth, COIs) are legion. Most policy researchers can readily provide many personal war stories from their professional experience. Generically, the most blatant cases tend to fall into four categories: 1. Investigators with a commercial or proprietary interest in the research outcome, or the use of funding from sources with a commercial or proprietary interest in the research outcome (e.g., Hilts, 2000; also see the 5 June 2002 special issue of JAMA). * Forthcoming in D. A. Moore, D. M. Cain,, G. Loewenstein, and M. Bazerman (eds.), Conflicts of Interest: Problems and Solutions from Law, Medicine and Organizational Settings. London: Cambridge University Press. ** Professor, Goldman School of Public Policy and Boalt Hall School of Law, University of California at Berkeley. Please send comments to MacCoun 3/21/2004 draft 2 2. The use of funding from sources with a political agenda that would benefit from particular research outcomes (e.g., Revkin, 2003). 3. Paid expert testimony in an adversarial legal proceeding (see Faigman, 1999). 4. The use of proprietary data sources unavailable to other investigators (Metcalf, 1998). To varying degrees, these four categories involve pecuniary motives and interests. But even if we could eliminate pecuniary motives in policy research, we would not eliminate conflicts of interest, because there is a fifth, more subtle category: 5. The influence, whether conscious or unconscious, of an investigator’s allegiance to extrapolitical or ideological values and attitudes. Even if pecuniary biases and ideological biases differ in their origins, they can have similar effects on research conduct and evidence interpretation. Pecuniary ideological biases are perhaps more troublesome in domains like medicine and engineering than in public policy, simply because there are few opportunities for researchers to reap financial reward for their research (patents, commercial applications). On the other hand, ideological biases may be less pervasive in domains like medicine and engineering, because disputes are more likely to center on means (the best techniques, author credit for innovations, etc.) than on ends (whether we should improve health, safety, or performance). In the public policy arena, the ends (income equality, reproductive rights, welfare entitlements, environmental preservation, a “drug-free society”) are often as contested as the means that would achieve them. Merriam-Webster's Collegiate Dictionary defines a conflict of interest as “a conflict between the private interests and the official responsibilities of a person in a position of trust.” In the domain of public policy research, we can construe both “private interests” and “a position of trust” either narrowly or broadly. Narrowly defined, “private interest” would involve the potential attainment of money, prestige, or other resources for oneself or one’s organization. A broader definition would include the researcher’s personal values and political views. Narrowly defined, “a position of trust” would involve particular professional offices with explicit rules proscribing bias or the pursuit of personal gain. A broader definition might invoke Robert Merton’s (1973) articulation of the norms of science that are widely shared in our culture: 1 And they may not, since ideological attitudes probably influence the seeking and receiving of biased pecuniary support. MacCoun 3/21/2004 draft 3 • Scientific accomplishments should be judged by impersonal criteria (“universalism”) rather than the personal attributes of the investigator. • Scientific information should be publicly shared (“communalism”). • Investigators should proceed objectively, putting aside personal biases and prejudices (“disinterestedness”). • And the scientific community to hold new findings to strict levels of scrutiny, through peer review, replication, and the testing of rival hypotheses (“organized skepticism”). There are now a good many published case studies documenting conflict of interest in this broader sense in many research domains, including HIV/AIDS (Epstein, 1996), tobacco (Cummings, Sciandra, Gingrass, & Davis, 1991; Glantz, 1996), sexual orientation (LeVay, 1996), intelligence testing (e.g., Fraser, 1995), drug prevention (Gorman, 2003; Moskowitz, 1993), risk prevention (Fischhoff, 1990), marijuana statistics (MacCoun, 1997), and global warming (Gelbspan, 1997). “Conflict of interest” can also be defined intrapersonally or interpersonally. In the traditional sense, the “conflict” is intrapersonal – a conflict between her role obligations and her behavior. But the term “conflict of interest” has also been used in a very different sense in the social psychology literature, one that defines the conflict interpersonally, between people or factions of people. For example, in the small group literature, McGrath’s (1984) group task circumplex defines “conflicts of interest” in terms of mixed-motive payoff structures. Of greater relevance to this essay, John Thibaut (a psychologist) and Laurens Walker (a lawyer) (1978) distinguish “cognitive conflicts,” where the parties have a joint interest in solving a problem, from “conflicts of interest,” where “a particular solution will maximize the outcome of one of the parties only at the expense of the other.” The intrapersonal definition is more conventional, but most real-world examples meet the interpersonal definition as well. Identifying the Right Normative System Thibaut and Walker (1978) identify the goal of cognitive conflicts as “truth,” and the goal of conflicts of interest as “justice.” These are lofty claims, but then, the authors had a lofty goal. Their work builds on a highly influential program of empirical research on citizen evaluations of 2 Koehler (1993) presents evidence that scientists endorse such norms. MacCoun 3/21/2004 draft 4 alternative conflict resolution procedures (Thibaut & Walker, 1975; also see Lind & Tyler, 1988; MacCoun, Lind & Tyler, 1992), but their theoretical goal is normative, not descriptive. They want to define the proper domains for inquisitorial vs. adversarial procedures of conflict resolution. In an adversarial process, as exemplified by the Anglo-American trial system, disputants retain “process control” by selectively presenting the facts most favorable to their position to a 3 party decision maker. In an inquisitorial process, evidence is assembled by the 3 party decision maker, or by a neutral investigator who reports to that decision maker. Some continental European legal systems are inquisitorial in this sense, but more relevant for present purposes, Merton’s norms of scientific practice are inherently inquisitorial. Thibaut and Walker make two normative claims. First, “an autocratic system delegating both process and decision control to a disinterested third party is most likely to produce truth,” hence cognitive conflicts should be resolved through the inquisitorial method. Second, “a procedural system designed to achieve distributive justice...will function best if process control is assigned to the disputants,” as exemplified by “the Anglo-American adversary model.” Thibaut and Walker’s normative theory has been much less influential than their empirical program, and although their treatment is more sophisticated and nuanced than this brief sketch, I don’t find it entirely persuasive. Though it is useful to examine ideal types in the laboratory, few real-world problems seem to fit neatly into these cognitive-conflict and conflict-of-interest bins. Thibaut and Walker allow for the possibility of “mixed conflicts,” but this category – arguably the largest category – they call for a mix of inquisitorial and adversarial procedures. Hence, I call attention to their theory because of the question they pose rather than the answers they offer. I believe that the central problem of conflict of interest in public policy research is the blurring of adversarial and inquisitorial norms and roles. Public policy research and the utilization of that research, falls far short of Merton’s inquisitorial ideals. Yet our allegiance to those norms, and our pretense to be operating under those norms, also keeps us from realizing some benefits of a more explicitly adversarial approach. We largely seem to muddle in the middle. In the abstract, a purely inquisitorial model might well be best, but we are unlikely to achieve one. An explicit, robust adversarial research process might be more attainable, and it might even have some advantages over a muddled mixed model, where some investigators play by one set of rules, some play by another, and some vacillate back and forth either strategically or unwittingly. But the adversarial model, whatever its merits in legal MacCoun 3/21/2004 draft 5 settings (and those are decidely mixed), has serious drawbacks outside the trial context. What are needed are clearer norms defining a realistically heterogeneous inquisitorialism. What Isn’t In This Essay The topic of bias in politically relevant research is an old one, and it has been examined many times before from other angles. There is an enormous literature on the details of bias in research methodology, including biased research designs, biased statistical analyses, biased data presentation, and experimenter expectancy effects (e.g., Rosenthal, 1994). Note that the kinds of investigator biases examined here may express themselves through these methodological problems, but bad methodology may instead reflect ignorance or real-world data constraints rather than bias on the part of the investigator. There is an extensive sociology literature on the effects of institutional factors, professional incentives, social networks, and demographic stratification on the scientific research process (see Cole, 1992; Zuckerman, 1988). And of course the troubled relationship between facts and values was a preoccupation of twentieth century philosophy of science (see Gholson & Barker, 1985; Laudan, 1990; Shadish, 1995). I also sidestep the postmodernist literatures on social constructivism, deconstructionism, hermeneutics, and the like, for reasons explained elsewhere (MacCoun, 1998, 2003; also see Gross & Levitt, 1994). ATTRIBUTING BIAS TO OTHERS It is very easy to attribute bias to researchers, and observers readily do so. But how are we to know whether the bias resides in the attributor, rather than (or in addition to) the investigator? The same forces that can produce bias in researchers can produce biases in consumers of that research. A case in point is the cottage industry in books denouncing “junk science.” These books are quick to criticize particular experts for sloppy and careless thinking – especially an overreliance on unsystematic and unrepresentative clinical case evidence and an underreliance on rigorous multivariate analysis and controlled experimentation. And what evidence do the authors offer for their indictment of junk science? Anecdotes about particular cases and particular experts, selected by an unspecified but surely non-random sampling process, with no correction for hindsight bias (the critiques make ready use of later science unavailable to the experts at the time in question), and no consideration of alternative motives for the expert MacCoun 3/21/2004 draft 6 testimony (MacCoun, 1995). I am not questioning whether sloppy or biased expert testimony occurs – it surely does – but simply arguing that we are often willing to attribute bias based on the “junkiest” of evidence. Obviously, some attributions of bias are self-serving; if an investigator presents findings you don’t like, the quickest way to discredit her – much quicker and more reliable than conducting your own study – is to question her motives or her integrity. But there are also some more subtle cognitive phenomena that complicate the attribution process. In a classic experiment by Jones and Harris (1967), students were enlisted to conduct an in-class debate on the topic of whether mid-1960s America ought to adopt a friendlier stance toward Fidel Castro. Half the participants were told that the debaters were assigned their positions by the debating coach, half were not told anything about how the debating roles were determined. After the debate, audience members estimated speakers' actual attitudes toward Castro. The audience overwhelming assumed the “pro-Castro” debater was indeed pro-Castro. This was true not only in the control condition, but among audience members who knew that the debating position was situationally determined (by the coach). Subsequent research using this “attitude attribution” paradigm has shown that this “shoot the messenger” tendency is quite robust (see Ross & Nisbett, 1991), exemplifing what Ross calls “the fundamental attribution error” – the tendency to give disproportionate weight to dispositional explanations (those internal to the actor, like traits and desires) for others’ behavior, while discounting or overlooking situational influences. The problem is a familiar one for policy analysts; audiences often assume that we favor (and, presumably, always favored) whichever political viewpoint our findings most readily benefit. In a recent statewide telephone survey of 1,050 California adults (MacCoun and Paletz, 2004), we found that for controversial social interventions -gun control, capital punishment, medical marijuana, and school vouchers (but not for healthy eating habits) -over half of all respondents were willing to speculate on a researcher's ideological beliefs knowing nothing other than whether a hypothetical study found that the intervention worked. For gun control and medical marijuana, positive findings led them to infer that the researcher was a liberal; for capital punishment, positive findings implied that the researcher was a conservative. Rather than viewing social science as an attempt to reveal facts about the world (the "discovery" model of MacCoun 3/21/2004 draft 7 research), many citizens construe social science as a process of political exhortation, and social scientists are seen advocates who find what they want to find.